docs: state the catalog trust boundary for the credential encryption key by LeeroyHannigan · Pull Request #228 · ExtendDB/extenddb

LeeroyHannigan · 2026-07-27T19:17:50Z

What

State the catalog trust boundary for the credential encryption key in the security model. The doc noted that the key lives in the catalog database and that a catalog backup includes it, but left the consequence to inference and did not mention that no rotation path exists.

Add the catalog database contents as an explicit trust boundary: the encryption key and the ciphertext it protects share one trust domain, so catalog read access yields every stored access key secret.
Scope that boundary to direct database access: the management API and console redact encryption_key from settings responses.
Record that there is no key rotation path; re-keying means re-creating the affected access keys.
Replace the "use a KMS-backed encryption layer at the PostgreSQL level" guidance with what extenddb does and does not do, and what protection below extenddb can and cannot achieve.
State that a catalog dump is credential-equivalent and should not be copied to less-protected environments.

Why

The security model understated a trust boundary. Making it explicit lets operators
protect the catalog database and its backups to the standard the credentials require.
#707 tracks keeping the key outside the catalog and rotating it.

Refs #707

Testing done

Documentation only, no code change:

Key storage location and unwrapped form: bootstrap_encryption_key in each backend's bootstrapper.rs (PostgreSQL: INSERT INTO settings (key, value) VALUES ('encryption_key', $1)).
encryption_key redacted from the settings responses of the management API and the web console: REDACTED_CONFIG_KEYS in crates/config/src/display.rs.
No rotation path: nothing in crates/ replaces encryption_key or re-encrypts the stored secrets under a new key. A grep for rotat finds only TLS certificates, backup data files, rotated access keys in the credential caches, and partition and stream label rotation.

Checklist

I have read CONTRIBUTING.md
All tests pass (cargo test --workspace)
Code is formatted (cargo fmt --check)
Clippy is clean (cargo clippy -- -W clippy::pedantic)
I have added or updated tests for new functionality
I have updated documentation if behavior changed
Breaking changes are noted below (if any)
If this changes the wire protocol, Storage trait, auth model, on-disk
format, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a, documentation only, no behavior change.

Breaking changes

None.

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

jcshepherd · 2026-08-05T21:52:17Z

 - **PostgreSQL security**: extenddb relies on PostgreSQL access controls and network security. Securing the PostgreSQL instance (firewall rules, TLS, authentication) is the operator's responsibility.
 - **Operating system security**: File permissions on `extenddb.toml`, TLS keys, and the PID file are the operator's responsibility.
- **Key management**: Access key secrets are encrypted with a locally generated AES key stored in the catalog database. For HSM-grade key management, use a KMS-backed encryption layer at the PostgreSQL level.
+- **Key management**: extenddb generates the credential encryption key locally and stores it unwrapped in the catalog database. It does not wrap the key with an external KMS or HSM, derive it from an operator passphrase, or rotate it. Deployments that require any of those must add protection below extenddb — for example PostgreSQL transparent data encryption, an encrypted volume, or a managed database with a customer-managed KMS key — which protects the key at rest but not from a principal that can query the catalog.


Maybe something for the backlog here: pluggable support for generating/managing credential encryption keys. Seems like this would be important for a lot of users.

Filed as #707: wrapping the key with a KMS or an HSM, a passphrase supplied at start-up, a pluggable key provider, and a rotation path that re-encrypts the stored secrets.

It also lists the questions an RFC would need to settle, such as what the server does when the provider is unreachable at start-up and how an existing deployment moves over.

jcshepherd

Thanks for documenting this. I think it'd be good to cut an issue to support other, more secure, mechanisms for handling the credential encryption key like the ones you mentioned: HSM, passcodes, etc. Status quo is probably not sufficient for many production uses.

The security model noted that the encryption key lives in the catalog database and that a catalog backup includes it, but left the consequence for the reader to infer and did not mention that no rotation path exists. - Add the catalog database contents as an explicit trust boundary: the key and the ciphertext it protects share one trust domain, so catalog read access yields every stored access key secret. - Scope that boundary to direct database access: the management API and console redact `encryption_key` from settings responses. - Record that there is no key rotation path; re-keying means re-creating the affected access keys. - Replace the "use a KMS-backed encryption layer at the PostgreSQL level" guidance with what extenddb does and does not do, and what protection below extenddb can and cannot achieve. - State that a catalog dump is credential-equivalent and should not be copied to less-protected environments.

jcshepherd · 2026-11-02T21:56:42Z

+- Encryption key generated during `extenddb init` and stored unwrapped in the catalog `settings` table, under the key `encryption_key`
+- The encryption key and the ciphertext it protects therefore share one trust domain: anyone who can read the catalog database can decrypt every stored access key secret. Restrict catalog database access to the extenddb service account, and protect catalog backups to the same standard as the credentials themselves.
+- The management API and web console redact `encryption_key` from settings responses. The key is readable only with catalog database access: a database client, or `extenddb settings get encryption_key --show-secret` on a host that holds the catalog connection string.
+- extenddb provides no key rotation path. Re-keying requires re-creating the affected access keys.


If I'm understanding correctly, this is too terse: it's not making the key rotation path, even if manual, very clear. This might be better:
Extenddb has no command to rotate the encryption key. Rotating it by hand makes every stored access key and session credential unreadable. Re-create all access keys using the new encryption key, and restart the server."

Taken, with session credentials added, and the restart now says why: the server keeps the old key and the decrypted secrets in memory.

jcshepherd · 2026-11-02T22:15:01Z

 1. **Client ↔ extenddb**: Untrusted. All input is validated. SigV4 signatures are verified. IAM policies are evaluated.
 2. **extenddb ↔ PostgreSQL**: Trusted network. The PostgreSQL connection string contains credentials. Use TLS for the PostgreSQL connection in production (`sslmode=require` in the connection string).
 3. **Admin ↔ Management API/Console**: Authenticated via admin credentials or IAM user credentials. CSRF tokens protect the web console.
+4. **Catalog database contents**: Credential-equivalent. The AES-256-GCM key that encrypts access key secrets lives in the catalog `settings` table alongside the ciphertext it protects, so read access to the catalog database yields every stored access key secret. Treat catalog read access, and any catalog backup, as equivalent to holding those credentials. See [Credential Storage](#credential-storage).


It might be important to note that on PostgreSQL, read access to the Catalog db gives you read access to the data_database_connection_string settings, which gives you log-in credentials directly to PostgreSQL, which essentially (if I understand those credentials correctly) gives you root-like permissions for ExtendDB. I'm not sure why we need unredacted db credentials to be stored in the db.

Added to the catalog item: on PostgreSQL the same table holds data_database_connection_string, which the console, the management API and the CLI mask, but a reader of the catalog database gets in full. Keeping it out of the catalog is a code change, so I filed #789 for it.

jcshepherd · 2026-11-02T22:18:31Z

+- The management API and web console redact `encryption_key` from settings responses. The key is readable only with catalog database access: a database client, or `extenddb settings get encryption_key --show-secret` on a host that holds the catalog connection string.
+- extenddb provides no key rotation path. Re-keying requires re-creating the affected access keys.
 - Console passwords hashed with bcrypt (cost factor 12)
 - No in-process credential cache — every request reads directly from PostgreSQL


This hasn't been true for a while: unencrypted credentials are in-memory with a running extenddb server.

Right. #786 rewrites this bullet with a section on the credential and policy caches, so I left this line to it, to keep the two PRs from conflicting. The new Server host item in Trust Boundaries says the running server keeps the key and recently used decrypted secrets in memory.

jcshepherd · 2026-11-02T22:21:35Z

 1. **Client ↔ extenddb**: Untrusted. All input is validated. SigV4 signatures are verified. IAM policies are evaluated.
 2. **extenddb ↔ PostgreSQL**: Trusted network. The PostgreSQL connection string contains credentials. Use TLS for the PostgreSQL connection in production (`sslmode=require` in the connection string).
 3. **Admin ↔ Management API/Console**: Authenticated via admin credentials or IAM user credentials. CSRF tokens protect the web console.
+4. **Catalog database contents**: Credential-equivalent. The AES-256-GCM key that encrypts access key secrets lives in the catalog `settings` table alongside the ciphertext it protects, so read access to the catalog database yields every stored access key secret. Treat catalog read access, and any catalog backup, as equivalent to holding those credentials. See [Credential Storage](#credential-storage).


We may also want to add that access to extenddb.toml is "credential-equivalent" as well.

Added as a fifth item, Server host: extenddb.toml holds the catalog connection string and the process memory holds the key and recently used secrets, so read access to the config file, a core dump or a memory read are credential-equivalent too.

jcshepherd

On second look, this could probably use another pass, partly because Things Have Changed. On that note, there are outdated statements elsewhere in the documentation:
10-security-model.md:236 says access key secrets "cannot be retrieved afterward". The new line 59 says a catalog reader can decrypt them all.
02-design-guide.md:40 and :129 put the key in an encryption_keys table. No backend has that table.
02-design-guide.md:131 repeats "no in-process cache for credentials".
docs/design/08-component-config.md:55, :124 and :432 describe an auth.encryption_key setting from EXTENDDB__AUTH__ENCRYPTION_KEY. AuthConfig has no such field (crates/config/src/lib.rs:246-256). A reader can think the key can be kept out of the catalog. All at base. Line 236 and the 02-design-guide.md lines are worth fixing in this PR, because they now conflict with it on the same topic.

- 10-security-model.md: the catalog trust boundary names the PostgreSQL data database login held in settings; a Server host boundary covers extenddb.toml and the server's memory; the rotation bullet says what a manual rotation breaks and why the server must restart; and the Operational Security bullet no longer says secrets cannot be retrieved. - 02-design-guide.md: the key is a settings row, not an encryption_keys table. - 08-component-config.md: drop the auth.encryption_key setting and EXTENDDB__AUTH__ENCRYPTION_KEY, which AuthConfig does not have.

LeeroyHannigan · 2026-11-03T09:55:03Z

@jcshepherd the new head merges main and addresses your second pass:

10-security-model.md: the catalog item now names the PostgreSQL data database login stored in settings; a new Server host item covers extenddb.toml and the server's memory; the rotation bullet uses your wording, with session credentials added; and the Operational Security bullet says a catalog reader can decrypt the secrets, instead of saying they cannot be retrieved.
02-design-guide.md: the key is a row in settings under encryption_key, and the encryption_keys table row is gone.
08-component-config.md: removed the auth.encryption_key setting and EXTENDDB__AUTH__ENCRYPTION_KEY, which AuthConfig does not have.
Left to docs(security-model): describe the credential and policy caches #786: the "No in-process credential cache" bullet and 02-design-guide.md:131, which docs(security-model): describe the credential and policy caches #786 rewrites, so the two PRs do not conflict.
Filed PostgreSQL: the data database login is stored in the catalog settings table #789 for keeping the data database login out of the catalog.

LeeroyHannigan requested review from amrith, c33howard, jcshepherd, pdf-amzn and yesyayen as code owners July 27, 2026 19:17

jcshepherd reviewed Aug 5, 2026

View reviewed changes

LeeroyHannigan added the v1 Needed for the 1.0 release label Oct 28, 2026

LeeroyHannigan mentioned this pull request Oct 31, 2026

Credential encryption key: keep it outside the catalog and add a rotation path #707

Open

LeeroyHannigan force-pushed the docs/catalog-trust-boundary branch from 1423ed9 to 1e67b26 Compare November 1, 2026 00:47

LeeroyHannigan requested a review from robinnsc as a code owner November 1, 2026 00:47

LeeroyHannigan requested a review from jcshepherd November 1, 2026 00:47

LeeroyHannigan force-pushed the docs/catalog-trust-boundary branch from 1e67b26 to 253d167 Compare November 1, 2026 01:13

LeeroyHannigan mentioned this pull request Nov 1, 2026

docs: back up and restore SQLite and PostgreSQL with their own tools #737

Open

5 of 8 tasks

jcshepherd reviewed Nov 2, 2026

View reviewed changes

jcshepherd requested changes Nov 2, 2026

View reviewed changes

robinnsc mentioned this pull request Nov 3, 2026

docs(security-model): describe the credential and policy caches #786

Merged

4 of 8 tasks

navareamol added this to the 1.0 milestone Nov 3, 2026

Merge branch 'main' into docs/catalog-trust-boundary

cac6eee

LeeroyHannigan mentioned this pull request Nov 3, 2026

PostgreSQL: the data database login is stored in the catalog settings table #789

Open

LeeroyHannigan requested review from eliseong, hitesh96db and navareamol as code owners November 3, 2026 09:54

LeeroyHannigan requested a review from jcshepherd November 3, 2026 09:55

Conversation

LeeroyHannigan commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing done

Checklist

Breaking changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcshepherd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcshepherd left a comment

Choose a reason for hiding this comment

Uh oh!

LeeroyHannigan commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LeeroyHannigan commented Jul 27, 2026 •

edited

Loading